The present invention relates to a technology for generating new videos by integrating a plurality of videos and, more particularly, a method of spatio-temporally integrating/managing a plurality of videos and a system for embodying the same and a recording medium for recording a program for carrying out the method.
With the progress of high performance of the computer, large capacity of the hard disk, reduction in size and digitization of the camcorder, and lower costs of them, spread of these devices into general families has been accelerated. According to such spread of the high performance computer and the mass storage hard disk and in addition establishment of the information compression technology, digitization of videos has become close to us. According to such digitization the videos, the videos have be able to be handled with the general-purpose personal computer and also be output to the high resolution display for the personal computer. In general, the videos have been limited to be output to the TV monitor (640xc3x97480 pixels) only, but this has enabled the videos to be output by the resolution in excess of the resolution of the TV monitor. For example, a plurality of videos has been able to be displayed simultaneously on the display for the personal computer, etc.
With such development of video devices, several methods of enhancing the videos have been reported. In a literature, Michael Irani and Samuel Peleg; xe2x80x9cMotion Analysis for Image Enhancement: Resolution, Occlusion and Transparencyxe2x80x9d, Journal of Visual Communication and Image Representation, Vol.4, No.4, December, pp.324-335, 1993, the method of achieving the high resolution of the videos by using motion information in the videos and the method of interpolating the background hiding area caused by the subject, etc. have been proposed. In addition, Laura A. Teidcio has reported the method of generating the high resolution still pictures from the videos (Patent Application Publication (KOKAI) Hei 5-304675). In recent, ORAD Corporation has published the system xe2x80x9cDigital Replayxe2x80x9d which can implement reproduction of the videos to which new values added are attached. As system functions, for example, the system has have the functions of enhancement representation, tracing, and magnification of the subject, and the function for displaying information of figures and segments thereof, distances and velocities, etc. together with the videos.
Moreover, there has been reported a new user interface of the videos. In a literature, xe2x80x9cA Magnifier Tool for Video Dataxe2x80x9d, Proceedings of CHI ""92, pp.93-98 (1992), M. Mill et al. have reported that the video frames are arranged in the space according to the resolution level of time, so that the new video looking and accessing style to the time can be attained from coarse time resolution to fine time resolution. Still more, in a literature, xe2x80x9cMotion Image Processingxe2x80x9d, Striking Possibilities, ADVANCED IMAGING, AUGUST (1992), E. Elliot and A. W. Davis have reported that the new representing method of time information of the videos and the intuitive access to the time information can be achieved by representing three-dimensional object, i.e., video picture (two-dimensional)+time (one-dimensional).
In the meanwhile, under circumstances in which it becomes easy to input the videos by the camcorder and display style can be made free, the request for handling of a plurality of videos has occurred. For example, xe2x80x9ca plurality of videosxe2x80x9d used herein signifies either the plurality of videos which are picked up by a plurality of camcorders set up in the sport stadium such as the Olympics or the videos which are picked up by a single camcorder but show behaviors of different players. Further, xe2x80x9chandlingxe2x80x9d used herein signifies that the plurality of videos are appreciated, compared, searched, edited, etc. simultaneously.
However, according to the above reports in the prior art, there have been unsolved problems in the plurality of videos. According to the reports in the prior art, the enhancement of one shot has been achieved, but the enhancement of the plurality of videos by applying relations between plural videos to the plurality of videos has not been implemented. In addition, the plurality of videos have not integrated spatio-temporally and managed integratedly. xe2x80x9cEnhancement of the plurality of videosxe2x80x9d used herein signifies to execute the correlation between the plurality of videos, to extract automatically the video contents information (object information reflected actually on the videos, i.e., subject information), and to represent them visually. Also, it signifies to generate one or plural videos in which correlation of the plurality of videos and the video contents information are represented visually. For instance, one example is to generate one video, in which respective players are overlapped in the common space, from the plurality of videos which pick up behaviors of different players. xe2x80x9cTo manage spatio-temporallyxe2x80x9d is that information extracted based on the spatio-temporal video structure is managed integratedly. For example, the subject and the background are separated and managed respectively and also their spatio-temporal relation information are managed. To manage the plurality of videos integratedly by using the spatio-temporal structure has not been included in the object of the prior art. It has been impossible to manage the plurality of videos spatio-temporally integratedly only by combining merely the prior arts.
In the prior art, the implemented video user interface has represented the videos again simply, so that it has not extracted positively information of the videos and relations between the plurality of videos to enhance the videos. Since the contents of the videos and the relations between the plurality of videos have not been taken into account, it has been impossible to grasp intuitively the videos and the contents of the videos and enhance newly the video information, especially, information concerning the time. Also, since the plurality of videos have not been managed spatio-temporally, it has been impossible to interact the contents of the videos more precisely.
Like the above, according to the reports as with the enhancement of the videos, the user interface, etc. reported in the prior art, there has been no enhancement to assign high value added to the plurality of videos and no representation which can manage, represent, and operate the videos spatio-temporally integratedly. In other words, there have been problems in the prior art.
It is an object of the present invention to provide a method of spatio-temporally integrating/managing a plurality of videos and a system for embodying the same and a recording medium for recording a program for carrying out the method, which are capable of integrating spatio-temporally a plurality of videos to enhance the videos and managing, representing, and operating spatio-temporally integratedly a plurality of videos, in order to acquire information of plural videos from a plurality of videos picking up the same space effectively in response to user""s interest and object in their own style.
In order to achieve the above object, the present invention provides a system for spatio-temporally integrating/managing a plurality of videos, comprising: an image data string memory portion for reading video data to save them as data string; a camcorder operation condition detecting portion for reading the data string from the image data string memory portion and detecting camcorder operation condition information including camcorder on/off information and camcorder operating information; a video splitting portion for splitting videos of the data string into respective shots based on the camcorder on/off information; a subject/background separating portion for separating a subject and a background every frame of the videos by using the camcorder operating information and physical feature properties; a subject motion information extracting portion for correlating subject information separated every frame between frames; a video taken space resynthesizing portion for resynthesizing video taken spaces from which the videos are picked up, based on the camcorder operating information and the background separated every frame; a shot-to-shot relation calculating portion for calculating spatial shot-to-shot relations between a plurality of video taken spaces, which are resynthesized by the video taken space resynthesizing portion respectively, based on a plurality of shots being separated; and a video structure information managing/storing portion for managing/storing separated subject information, correlated subject information, the camcorder operation condition information, background information, and shot-to-shot relation information.
Also, the above system for spatio-temporally integrating/managing a plurality of videos, further comprises: a video structure information transmitter/receiver for transmitting or receiving all or a part of extracted subject information, the camcorder operation condition information, the background information, the shot-to-shot relation information, and the video data.
Similarly, the above system for spatio-temporally integrating/managing a plurality of videos, further comprises: a resynthesizer for resynthesizing one or plural video taken spaces and one or plural subjects based on information stored/managed in the video structure information managing/storing portion in compliance with one or both of predetermined conditions and user""s requests; a display unit for displaying resynthesized videos in the resynthesizer; a user input portion for inputting the user""s requests for resynthesis based on the videos which are displayed on the display unit; and an output portion for outputting the videos which are displayed on the display unit to an external device in a digital or analogue manner.
In order to achieve the above object, the present invention provides a method of spatio-temporally integrating/managing a plurality of videos, comprising: an image data string saving step of reading video data to save the video data as data string in an image data string memory portion; a camcorder operation condition detecting step of reading the data string from the image data string memory portion and detecting camcorder operation condition information including camcorder on/off information and camcorder operating information; a video splitting step of splitting videos of the data string into respective shots based on the camcorder on/off information; a subject/background separating step of separating a subject and a background every frame of the videos by using the camcorder operating information and physical feature properties; a subject motion information extracting step of correlating subject information separated every frame between frames; a video taken space resynthesizing step of resynthesizing video taken spaces from which the videos are picked up, based on the camcorder operating information and the background separated every frame; a shot-to-shot relation calculating step of calculating spatial shot-to-shot relations between a plurality of video taken spaces, which are resynthesized by the video taken space resynthesizing step respectively, based on a plurality of shots being separated; and a video structure information managing/storing step of managing/storing separated subject information, correlated subject information, the camcorder operation condition information, background information, and shot-to-shot relation information.
Also, the method of spatio-temporally integrating/managing a plurality of videos, further comprises: a video structure information transmitting/receiving step of transmitting or receiving all or a part of extracted subject information, the camcorder operation condition information, the background information, the shot-to-shot relation information, and the video data.
Similarly, the method of spatio-temporally integrating/managing a plurality of videos, further comprises, after the video structure information managing/storing step: a resynthesizing step of resynthesizing one or plural video taken spaces and one or plural subjects based on stored/managed information in compliance with one or both of predetermined conditions and user""s requests; and a display or outputting of displaying or outputting videos resynthesized by the resynthesizing step.
In order to achieve the above object, the present invention provides a computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos, comprising: an image data string saving step of reading video data to save the video data as data string in an image data string memory portion; a camcorder operation condition detecting step of reading the data string from the image data string memory portion and detecting camcorder operation condition information including camcorder on/off information and camcorder operating information; a video splitting step of splitting videos of the data string into respective shots based on the camcorder on/off information; a subject/background separating step of separating a subject and a background every frame of the videos by using the camcorder operating information and physical feature properties; a subject motion information extracting step of correlating subject information separated every frame between frames; a video taken space resynthesizing step of resynthesizing video taken spaces from which the videos are picked up, based on the camcorder operating information and the background separated every frame; a shot-to-shot relation calculating step of calculating spatial shot-to-shot relations between a plurality of video taken spaces, which are resynthesized by the video taken space resynthesizing step respectively, based on a plurality of shots being separated; and a video structure information managing/storing step of managing/storing separated subject information, correlated subject information, the camcorder operation condition information, background information, and shot-to-shot relation information.
Also, the computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos, further comprises: a video structure information transmitting/receiving step of transmitting or receiving all or a part of extracted subject information, the camcorder operation condition information, the background information, the shot-to-shot relation information, and the video data.
Similarly, the computer readable recording medium for recording a program for spatio-temporally integrating/managing a plurality of videos, further comprises, after the video structure information managing/storing step; a resynthesizing step of resynthesizing one or plural video taken spaces and one or plural subjects based on stored/managed information in compliance with one or both of predetermined conditions and user""s requests; and a display or outputting of displaying or outputting videos resynthesized by the resynthesizing step.
According to the present invention, video data are read, then saved image data string are read out, then camcorder operation condition information including camcorder on/off information and camcorder operating information are detected, then videos are split into respective shots based on the camcorder on/off information, then the subject and the background are separated frame by frame by using the camcorder operating information and physical feature properties, then subject motion information are extracted by correlating separated subject information between frames, then the video taken spaces are resynthesized based on the camcorder operating information and the background every frame, then spatial shot-to-shot relations are calculated between a plurality of pick-up spaces which are resynthesized based on a plurality of shots respectively, and above resultant information are managed/stored. As a result, spatio-temporal integration of a plurality of videos can be achieved which enables the spatio-temporal enhancement of the plurality of videos, spatio-temporal and integrated management, representation, and operation.
By transmitting or receiving all or a part of extracted subject information, etc., the user can select, search, or pick up such information freely at any location or transmit them to any users located at any location. In addition, any user can send such information into the system from any location.
In addition, since one or plural video taken spaces and one or plural subjects are resynthesized, displayed, or output to an external device in analogue/digital manners based on the above managed and stored information in compliance with the predetermined conditions and the user""s request, the users can acquire information of plural videos from a plurality of videos picking up the same space simultaneously, intuitively, and effectively in response to their interest and object in their own style.